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Abstract 

We consider chains of random constraint satisfaction models that are 
spatially coupled across a finite window along the chain direction. We 
investigate their phase diagram at zero temperature using the sur- 
vey propagation formalism and the interpolation method. We prove 
that the SAT-UNSAT phase transition threshold of an infinite chain is 
identical to the one of the individual standard model, and is therefore 
not affected by spatial coupling. We compute the survey propagation 
complexity using population dynamics as well as large degree approx- 
imations, and determine the survey propagation threshold. We find 
that a clustering phase survives coupling. However, as one increases 
the range of the coupling window, the survey propagation threshold 
increases and saturates towards the phase transition threshold. We 
also briefly discuss other aspects of the problem. Namely, the conden- 
sation threshold is not affected by coupling, but the dynamic threshold 
displays saturation towards the condensation one. All these features 
may provide a new avenue for obtaining better provable algorithmic 
lower bounds on phase transition thresholds of the individual standard 
model. 
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1 Introduction 



The field of modern error correcting codes used on noisy communication 
cliannels lias witnessed an interesting recent development. Spatially coupled 
Low-Density Parity-Check (LDPC) codes, initially introducecll] by Felstroem 
and Zigangirov pQ, have been recognized to have excellent performance due 
to the threshold saturation phenomenon [2]. We refer to [3] for the history 
and review of contributions in the field of communications, and a general 
analysis of the phenomenon. 

Recently we introduced an elementary statistical mechanical model, namely 
a chain of coupled Curie- Weiss spin systems [5], that already captures 
all the main features of this phenomenon, and shows that it is related to 
basic concepts of statistical mechanics. In a nutshell, this model is a one- 
dimensional chain of complete graph Ising models coupled in the longitudinal 
directioiiH by a Kac-like potential. When the range of the Kac potential goes 
to infinity (and its intensity to zero) the spinodal curve is pushed towards the 
coexistence one. The stable phase undergoes nucleation and grows, starting 
from the ends of the chain, and consequently (in the Kac limit) the metasta- 
bility domain disappears from the phase diagram. This has important algo- 
rithmic consequences in the context of error correcting codes, for the recovery 
of an original message from the corrupted one. 

We already argued in [1], [5] that threshold saturation occurs quite gen- 
erally when mean field models are coupled together into a one-dimensional 
chain and the longitudinal range of the coupling increases to infinity. The in- 
dividual mean field model may be some sort of spin glass system on a sparse 
Erdoes-Renyi like random graph or on a complete graph (or on an hyper- 
graph). The sparse case is relevant to error correcting codes; see [5] and 
references therein. The case of complete graphs is relevant to compressive 
sensing, another topic to which these ideas have been successfully applied 

[8], [9], m- 

Other models, defined on sparse random graphs, that are of great interest 
both in theoretical computer science and statistical mechanics, are random 
Constraint Satisfaction Problems (CSP). We refer the reader who is not fa- 
miliar with such problems to the recent book [11] . In this paper we investigate 
random spatially coupled-CSP. We specifically concentrate on two main rep- 
resentatives: satisfiability (SAT) and coloring (COL). We also briefly discuss 
the XOR-SAT problem which is somewhat similar to the LDPC codes on the 
binary erasure channel, but has an interest of its own. 

^In its original form the construction goes under the name Terminated Convolutional 
Low-Density Parity-Check codes. 

^Similar models have already been considered in other contexts [5], [?]■ 



2 



Here we focus on the zero-energy states of CSP. There are two ways 
to formulate the problem. One can directly minimize the Hamiltonian, or 
one can study the uniform measure over zero-energy states. We will focus 
essentially on the first aspect. The second one will only be briefly discussed. 
We say that a SAT (resp. UNSAT) phase corresponds to a vanishing (resp. 
finite) average ground state energy per variable, in the thermodynamic limit. 
Since the ground state energy per variable concentrates, this means that 
in a SAT (resp. UNSAT) phase there are, with high probability, at most 
a sub-linear (resp. at least a linear) number of unsatisfied constraints. In 
the language of computer science the problems that we are investigating are 
randomized versions of MAX-SAT and MAX-COL. 

Coupled-CSP are based on chains of L individual landom bipartite graphs 
with constant degree K for constraint nodes, and Poisson degree with mean 
aK for variable nodes. Each individual bipartite graph is appropriately 
coupled to its neighbors across a window of size w. The precise construction 
is explained in detail in section |2l The number a > is a measure of the 
constraint density and plays the role of a control parameter. In the K- 
SAT problem each constraint corresponds to satisfying a disjunction of K 
literals. In Q-COL, we have K = 2 and all variable nodes connected to 
the same constraint node must have different colors in a Q-ary alphabet. 
Despite the similarities in construction with the LDPC case, in general, CSP 
(and coupled-CSP) are considerably more difficult to analyze. To study 
the ground state problem we adopt the Survey Propagation (SP) formalism, 
which is derived from the zero-temperature (level-1) cavity method of spin- 
glass theory [12]. We refer to [11] for a recent pedagogical account, but for 
the convenience of the reader we review and adapt the formalism to coupled 
CSP, in a streamlined form, in appendix [B] 

Let us pause and explain the predictions of the SP formalism for indi- 
vidual graph ensembles [13], [H]. SP is a sophisticated mean field theory 
based on a set of fixed point equations. They predict the existence of a 
SAT-UNSAT phase transition when a crosses a critical threshold as- At a 
lower value asp one finds a bifurcation from trivial to non-trivial solution of 
the fixed point equations. In the interval [asp, a^] the solution space is frag- 
mented in an exponentially large (in system size) number of well separated 
clusters of SAT ground states in Hamming space (binary or Q-ary). The rate 
of growth of the number of such clusters with system size, is called the zero- 
energy complexity and is positive in the interval [asPi^s]- The complexity 
becomes formally negative at a^. 

We consider the SP equations for coupled i^-SAT and Q-coloring models 
and solve them by the method of population dynamics (sections [3] and H]). 
We find a positive complexity in an interval [asp,L,«,, Ois,L,w] which allows to 
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determine the SAT-UNSAT phase transition point as^L,w (where the com- 
plexity becomes formally negative). We make the following observations for 
the interval where the complexity is positive. We have that as,L,w > o^s and 
o^s,L,w i cts as L increases (and w fixed). Interestingly we find that threshold 
saturation takes place, namely asp,L,w — ?■ as L and w both increase such 
that 1 << w << L. These findings are supported by a large K and Q 
analysis of the SP fixed point equations of coupled CSP (sections [3] and S]). 
In this limit the fixed point equations reduce to one-dimensional difference 
equations, analogous to the ones found for the Curie- Weiss chain or coupled 
LDPC codes on the binary erasure channel. This allows to study an "average 
total warning probability" that characterizes the phase of the system. This 
quantity is somewhat analogous to the average magnetization in the CW 
chain, or the average erasure probability for LDPC codes. A corresponding 
"van der Waals curve" displays an oscillating structure around a "Maxwell 
plateau". Each oscillation corresponds to a state of the system character- 
ized by a kink profile for a "local warning density" and a "local complexity 
density" along the chain. 

The thermodynamic limits of the average ground state energies per node, 
for the chain and the individual ensembles are proven to be equal (section 
[5]). The proof uses an interpolation method [15], [16], [T7] in a convenient 
combinatorial form similar to [18]. This result is of some importance because 
it establishes that non-analytic points in the average ground state energy per 
node of the chain and individual ensembles, occur at the same constraint 
density. In other words one must have lim^^+oo as,L,«; = «s- 

In sections |6] and [7] we briefly discuss further important aspects that will 
be the object of more detailed future work. 

The SP formalism says nothing about the relative sizes (internal entropy) 
of clusters of solutions and does not take into account which of them are 
"relevant" to the uniform measure over zero energy solutions. This issue 
is addressed by the entropic cavity method [19], [20], [21], which allows to 
compute the so-called dynamical and condensation thresholds ad and «£• We 
have computed the dynamical ad,L,w and condensation ac,L,i« thresholds of 
coupled CSP, and observe that as L increases limi_>._,.oo ac,L,i« — > Q^c fixed) 
while ad^L,w etc when both w and L increase in the regime 1 « w « L. 
The first observation is consistent with a rigorous result proved in appendix 
Rl the thermodynamic limit of the free energy (at finite temperature) of the 
chain is identical to that of the individual model. These issues are discussed 
in section O 

This work may have interesting algorithmic consequences. Any bound 
on as,L,w can be turned into a bound for Og by taking L — )■ +oo (note this 
is also true for the condensation threshold). In particular, algorithmic lower 
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bounds on as^L,w can be turned into lower bounds for a^. Now, because of the 
saturation of the SP and dynamical thresholds of coupled CSP, the values of a 
for which the space of zero-energy solutions is fragmented into well separated 
clusters, are substantially larger, compared to values of individual ensembles. 
Therefore one may hope that a form of algorithmic threshold saturation, or at 
least algorithmic threshold increase, happens for some well chosen algorithms 
applied to coupled CSP. This may allow to prove better algorithmic lower 
bounds on as,L,w and thus as- The proposed methodology is briefly discussed 
and illustrated in section [7] with simple peeling algorithms. 



We define a general class of CSP that form the individual ensemble. Then 
we couple these, to form one-dimensional chains called spatially coupled- CSP 
ensembles. 



First, we specify an ensemble (A^, K, a) of random bipartite graphs. Let 
G = (y U C,E) with variable nodes i ^ V , constraint nodes c G C and 
edges {c,i) connecting sets C and V. We have \V\ = N, \C\ = M, where 
M = \_aN\ (the integer part of aM) and a is a fixed number called the 
constraint density. We call the size of the graph which is to be thought 
as large, — +00. All constraints c have degree K, and each edge (c, i) 
emanating from c is independently connected uniformly at random (u.a.r.) 
to a node in i E V. As A^ — )■ +00, the degrees of the variable nodes tend to 
independent identically distributed (i.i.d.) with distribution Poisson(a_ft'). 

We denote by di the set of constraints connected to variable node i and 
by do the set of variable nodes connected to a constraint c. 

For each graph G of the ensemble [A^, K, a] we define a Hamiltonian (or 
cost function). To the variable nodes i E V we attach variables Xj G A" taking 
values in a discrete alphabet X. To each constraint c G C we associate a 
function ipdxgc) which depends only on the variables xqc = (a^i)ie(9c connected 
to c. For constraint satisfaction problems ipdxdc) G {0, 1}; we say that the 
constraint is satisfied if ipc{xdc) = 1 and not satisfied if ipc{xdc) = 0. The 
total Hamiltonian is 



For many problems of interest the functions ipc are themselves random. This 
will be made precise in each specific example; the only important condition 



2 General Setting 



2.1 Individual CSP ensemble [N,K,a]. 




(1) 
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is that the functions ipc are i.i.d. for all c G C The ground state energy is 
min^'H{x), the minimum possible number of unsatisfiable constraints. Our 
main interest is in the average ground state energy per node 

e^ia) = ^E[mmn{x)] (2) 

iV X 

where the expectation is taken over the [A^, K, a] ensemble and possibly over 
the randomness of ipc- 



2.2 Coupled-CSP ensemble [N, K,a,w,L]. 

This ensemble represents a chain of coupled underlying ensembles. Figure 
[T] is a visual aid but gives only a partial view. We align positions 2 G Z. 
On each position 2; G Z, we lay down variable nodes labeled {i, z) G V^, 
i = 1, ■ ■ ■ , N . We also lay down M = [aN\ check nodes labeled (c, z) G C^, 
c = 1, ■ ■ ■ , M. When the node labels are used as subscripts, say as in a(i_^) or 
a(c,2), we will simplify the notation to aiz or Qcz- Let us now specify how the 
set of edges, E, is chosen. Each constraint (c, z) has degree K, in other words 
K edges emanate from it. Each of these K edges is connected to variable 
nodes as follows: we first pick a position z + k with k uniformly random in the 
window {0, ■ ■ ■ , w—1}, then we pick a node {i, z+k) u.a.r. in V^+k, and finally 
we connect (c, z) to {i, z + k). The set of edges emanating from {i, z) can be 
decomposed as a union U^rQ^{((c, z — k), {i, z)) \ c E Cz]- Asymptotically as 
A^ — +00, its cardinality is Poisson(aii'); and the cardinalities of each set 
in the union are i.i.d. Poisson(^). 

Finally, take L an even integer. Restrict the set of constraint nodes to 
U^^_L_,_]^ ... j^lCz and delete edges emanating from constraints that do not 
belong to this set. Restrict the set of variable nodes to ^z=-i^+i,- ,i^+w-i 

As in subsection 12. ![ we have a set of variables Xiz G X and constraint 
functions ipczi^dcz) taking values in {0,1}. To each coupled graph in the 
ensemble we associate the Hamiltonian depending on x = (xiz), {i,z) G 

^z=-^+l,-,^+w-lVz, 

L 
2 

"Hcoul^) = ^ ^{.'^-'ipcz{xa(^cz)))- (3) 

The minimum over x_ is the ground state energy and its ensemble average per 
node is ^ 

eN,L,w{a) = —"^[vcvmUconix)], (4) 
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Figure 1: A representation of the geometry of the graphs with window size w = 3 
along the "longitudinal chain direction" z. The "transverse direction" is viewed 
from the top. At each position there is a stack of N variable nodes (circles) and 
a stack M constraint nodes (squares). The depicted links between constraint and 
variable nodes represent stacks of edges. 

where E is over the [A^, K, a, w, L] graph ensemble and on the randomness in 

Remark about the boundary conditions. In the formulation above we 
have free boundary conditions. However, the average degree of the variable 
nodes close to the boundaries is reduced so that the CSP is easier close to the 
boundaries. Variable nodes close to the right boundary z = ^ + 1, ^+w — l 
have degrees Poisson(^(-| + w — z)), and those close to the left boundary 
z = + 1, + w — 1 have degrees Poisson(^(2 + ^)). It is sometimes 
convenient to imagine that the boundary nodes are connected to "satisfied 
extra constraint nodes", and all have Poisson(Q;i^') degree. 

2.3 K- Satisfiability and Q-Coloring. 

We define the main examples of constraint satisfaction problems that we 
analyze in this paper. 

The K- SAT problem. The individual system is defined as follows. We take 
Xi G {0, 1} the Boolean alphabet. Set n{xi) = Xt for the negation operation, 
and define n'^{xi) = Xi when d = and n'^{xi) = n{xi) = Xi when d = 1. Pick 
Bernoulli(l) i.i.d. numbers d(^c,i) for each edge (c, i) G E. We say that an 
edge is dashed when d(^c,i) = 1 and full when d(^c,i) = 0. With this convention, 
a variable in a constraint is negated when it is connected to a dashed edge, 
and is not negated when it is connected to a full edge. We set 

M^ac) = l(V,ea,(n'^<=>') (x,)) = 1). (5) 

These definitions are extended to the coupled system in an obvious way 

i^cz{xe(cz)) = l(Vi„ea(c^)(ri'^<"''">(a;m)) = 1), (6) 
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where the important point is that d(^cz,iu) are i.i.d. BernouUi(i) for all edges. 
The ground state energy counts the minimum possible number of unsatis- 
fiable constraints. The instance is satisfiable iff the ground state energy is 
equal to zero. 

The Q-COL problem. For the individual ensemble, we take Xi & X = 
{0, • • ■ ,Q — 1} the Q-aij color alphabet, K = 2 for the constraint node 
degrees, and 

ipc{xdc) = ^ Xj for {i,j} = (9c). (7) 

Since the constraints have degree 2 one can replace them by edges connecting 
directly i and j for i,j G dc. The induced graph is, in the large size limit, 
equivalent to the Erdoes-Renyi random graph G{N, ^) ). The constraint ([7]) 
forbids two neighboring nodes to have the same color. 

These definitions are easily extended to the coupled system. The induced 
graph (obtained by replacing constraints by edges) is now a coupled chain of 
Erdoes-Renyi graphs. In place of ([7]) we take Xiz ^ X = {0, ■ ■ ■ ,Q — 1} and 

i^cz{xd(^cz)) = i{xiu 7^ Xjv for {{i,u), = d{c,z)). (8) 

Given an instance of the induced graph, the ground state energy counts the 
minimum possible number of edges with vertices of the same color. The 
graph is colorable iff this number is zero. 

The K-XORSAT problem. We briefly give relevant definitions that will 
be used in section [7] and appendix \K\ For the individual system Xj G {0, 1} 
and ipc{.Xdc) = ^{.®i£dcXi = b^) with b^ being i.i.d. Bernoulh(|). Similarly 
for the coupled system ipcz{xd{cz)) = '^i®iu&d{cz)Xiu = bcz) with bcz being i.i.d. 
Bernoulh(|). 

2.4 Static phase transition threshold 

For the purpose of analysis, it is useful to also consider an ensemble of coupled 
graphs with periodic boundary conditions. This ensemble is simply obtained 
from the [N^K.,a^w,L\ ensemble by identifying the variable nodes {i^z) at 
positions z = ^ + k with nodes {i^z) at positions z = + k for each 
k = 1, ■ ■ ■ ,w — 1. The formal expression of the Hamiltonian H^ou (x) is the 
same as in (|3]) except that now x = (xiz) with U^^_l_,_j^ ... l Vz. Quantities 
pertaining to this ensemble will be denoted by a superscript "per". 

Theorem 1 (Comparison of open and periodic chains). For the general 
coupled-CSP [N,K,a,w,L\ ensembles we have 

ccw 

\eN,LA(^) - (^N'L,wi(^)\ < -J^- (9) 
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This theorem has an easy proof given in section O 

The next theorem does not have a trivial proof and is stated here for the 
special cases of K-SAT and Q-COL. While it is presumably valid for many 
other CSP's, we do not expect that it should hold in complete generality. 
For example it is well known from models in other areas of statistical and 
condensed matter physics that the ground states of a periodic chain can break 
translation invariance (e.g crystals develop non-trivial periodic patterns) and 
then have lower energy than the homogeneous ground state. If this happens 
the statement of the theorem cannot possibly hold. 

Theorem 2 (Thermodynamic limit). For the K-SAT and Q-COL models 
the two limits limAr^+oo eAr(a) and lim7v-i.+oo tL,(tt) exist, are continuous, 
and non- decreasing in a. Moreover they are equal, 

lim lim e^J\ ,„(a) = lim cnM. (10) 



Remark about XORSAT. We prove such a theorem for K-XORSAT with 
K even in appendix |Al The proof breaks down for K odd, although the result 
is presumably true in that case also. 

Standard methods of statistical mechanics [2S] do not allow to prove 
the existence of the limits because the underlying graphs have expansion 
properties. When the system is cut in two parts the number of edges in the 
cut is of the same order as the size of the two parts and is not just a "surface" 
term. Therefore sub-additivity of the free and ground state energies become 
non-trivial. However, interpolation methods allow to deal with this issue. 
The existence of the limit for limAr^+oo eAr(a), as well as the fact that the 
function is continuous and non- decreasing, is proved for a range of models 
including the present ones in [18], and it is easy to see that the same sort 
of proof works for the periodic chain. This proof will not be repeated. In 
section we provide the proof for the equality of the two limits. This is again 
based on two interpolations which provide upper and lower bounds. Note 
that concentration of the ground state and free energies is also implied by 
standard arguments not discussed her^. 

We are interested in the thermodynamic limit 

lim = lim lim 

therm L— >+oo A^— ^+00 

■^However concentration of the number of solutions in the SAT phase is more subtle see 
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for the open chain, which captures the regime of a long one- dimensional 
coupled-CSP. From theorems [T] and [2] we deduce that 



lim eAr,L,«;(a) = lim e^'''^^(a) = lim eAr(a). (11) 

therm therm ' ' N—>+oo 

Since the energy functions are non- decreasing we can define a natural "static 
phase transition" threshold as follows. 

Definition 1 (Static phase transition threshold). We define 
as,L,w = sup{a\ lim eN,L,w{a) = 0} = sup{a\ lim e^""^ ^(a) = 0}, (12) 

A''— >-+oo Af— >+oo ' ' 

and 

as = supjal lim cjyx.wici) = 0} = sup{a| lim e^/,^(«) = 0} 

therm ' ' therm ' ' 

= sup{a| lim eAr(a) = 0}. (13) 

The supremums in the first definition are equal because of theorem |2] 
and those in the second definition are equal because of (ITT]) . Note also 
that limL^+oo (^s,L,w = «s- The definition of as implies that, for a given 
instance, when a < as (resp. a > ag) the number of unsatisfied constraints 
is o{N) (resp. 0{N)) with high probability. However it is not known how 
to automatically conclude that a fixed instance is SAT (resp. UNSAT) with 
high probability when a < as (resp. a > as). 

Remark about finite temperatures. The theorems of this subsection 
have finite temperature analogs presented in appendix [A] As explained in sec- 
tion[6]these suggest that the condensation threshold obeys limi^^oo C(c,l,w = 
a.. 



2.5 Zero temperature cavity method and survey prop- 
agation formalism 

We briefly summarize the simplest form of the cavity method and survey 
propagation equations for the coupled-CSP. More details on the formalism are 
presented in appendix [Bl When the graph instance is a tree, the minimization 
of ([3]) can be carried out exactly. This leads to an expression for min^7^cou(ai) 
in terms of energy-cost messages Eiu^cz{xiu) and Ecz~^iu{.Xiu) that satisfy 
the standard min-sum equations (see equ. fll36p and (11371) ). These mes- 
sages are normalized so that miuj.-^ _E'j„_^c2(a^i?i) = "^^"^Xiu^cz^iuixiu) = 
and they take values in {0, 1}. They may be interpreted as warning mes- 
sages. Roughly speaking, nodes inform each other on the most favorable 
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values that the variable Xiu should take in order to avoid energy costs. The 
ground state energy (on the tree) is given by the Bethe energy functional 
£[{Eiu (.)}] (see equ. fll39p ). For a general graph instance one 

considers the Bethe energy functional fll39p as an "effective Hamiltonian" 
and studies the statistical mechanics of this effective system. The min-sum 
equations are the stationary point equations of this functional and the set of 
solutions {Eiu^czi-) , Ecz-^iu{-)} characterize the state of the system. 

It turns out that the min-sum equations may have exponentially many (in 
system size) solutions with infinitesimal Bethe energy per node as N ^ +00. 

A solution {-Ej-^L^^(.), -E^zLiMl-)} with infinitesimal Bethe energy defines a 
pure Bethe statqj denoted by the superscript (p). We define the average 
zero-energy complexity as 

1 

«)(«)= lim lim ——E[ln(number of states p with —— = e)]. (14) 

This quantity counts the number of pure Bethe states. The typical behavior 
of the complexity as a function of a is as follows. Below an SP threshold it 
vanishes, then jumps to a positive value and decreases until it becomes neg- 
ative at the static phase transition threshold. It therefore allows to compute 

asp,L,«; = ird{a\LL,w{a) > 0}, (15) 

as,L,w = sup{a|Si^^(a) > 0}. (16) 

It is a feature of the cavity theory that the static phase transition thresholds 
defined according to the energy (1121) and complexity fll6p coincide. 

The complexity is the Boltzmann entropy (on the zero energy shell) of the 
effective statistical mechanical problem with Hamiltonian 
£[{Eiu^cz{-), Ecz^iui-)}]- It turns out that this can be computed, thanks 
to an effective partition function on the same sparse graph instance, again 
within a message passing formalism. In this context messages are called 
surveys. They count the fraction of pure Bethe states with given warning 
messages. Surveys Qiu^cz{Eiu^cz{-)) and Qcz^iu{Ecz^iu{.)) are exchanged 
between variable and constraint nodes according to survey propagation equa- 
tions (see fll44p and fll45p ). The average complexity (IT^ can be computed 
by a Bethe type formula for the entropy of the effective model. 

The survey propagation equations fll44p . fll45p allow to compute the dis- 
tribution over pure Bethe states, of the vectors {Ecz^iu{xiu), Xiu G X). These 
are | A* | -component vectors with components in {0, 1}. Thus the surveys are 

^We adopt this terminology to make a distinetion with the mathematicahy precise 
notion of pure state for usual Ising models . 
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supported on an alphabet of size at most 2' L Often the effective size of the 
alphabet is smaller (it is lA:"! + 1 in the specific problems considered here) 
because the warning propagation equations fll36p . fll37p restrict the possible 
values of {Ecz^iu{xiu),Xiu G X). This simplification is used for each model 
separately in the next sections. 

Let us summarize the main observations that follow from the detailed 
analysis in sections [3] and HI As L — t- +oo, we find that the complexity curves 
^L,w{c() supported on the interval [asp,L,w, C(s,l,w] converge to a limiting curve 
supported on the limiting interval [asp.«,,as]. Moreover, on this later 
interval, T,w{a) coincides with the complexity S(q;) of the individual system 
{L = w = 1). This is illustrated on Figure [2l We observe that as,L,w tends to 
as from above. Also for moderate L one generally has asp^L,w > ^s? but this 
inequality is reversed for L large enough, and lim^^+oo asp,L,w = asp.w < 

We observe the threshold saturation, namely lim^^+oo asp,w t (^s- In fact 
we expect (from [5]) that the gap |q;sp,u; — ctsl is exponentially small in w {K 
fixed) but this is hard to assess numerically. One also observes that for w 
fixed the gap increases with increasing K. 

"zero energy complexity 




Figure 2: Complexity of the individual ensemble S(a) (i.e. L = w = I) and 
limiting complexity S«,(a) of the coupled ensemble for L — )• +oo. We have asp.ui — ^ 
Os as w ^ +00. 

We point out that the complexity of the chain with periodic boundary 
conditions converges to that of the individual system in the infinite length 
limit. In other words there is no threshold saturation as long as the boundary 
conditions are periodic. This is easily understood by realizing that the sur- 
vey propagation equations are purely local and have a translation invariant 
solution when the boundary conditions are periodic. 

Finally, let us mention that we observe similar features for the entropic 
complexity curve. In this case ad plays the role of asp and ac that of as- We 
have ad,w — ^ ^c- In particular, limi^+oo ac,L,i« = «c and 
lim^^_i_Qo 

lirai^+oo Cid,L,w = Oic (see section [6]). 
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3 Coupled K-SAT Problem 



3.1 Numerical implementation 

We begin with a convenient parametrization of the messages (see e.g [TTj). 
Since X = {0, 1}, the warning (energy costs) messages are two-component 
vectors (Eiu^cziO) , Eiu^czi^)) and (i?c2^m(0), -E'c^^i„(l)) which take three 
possible values (0, 1), (1,0) and (0,0). Warning (0, 1) means that Xiu should 
take the value 0, warning (1, 0) means that Xiu should take value 1, and warn- 
ing (0, 0) means that Xiu is free to take any value. Messages from variables 
to constraints can be conveniently parametrized as follows, 

^5 ^ \Qiu^cz{0, 1) if is negated in cz, 
lu^cz I g,.^^^^(i^ 0) if Xiu is not negated in cz. 



This is the fraction of pure states for which the variable is forced to satisfy 
the constraint. Similarly, 



Q 



u 

iu—^cz 



j(0, 1) if Xiu is not negated in cz, 
j(l, 0) if Xiu is negated in cz. 



This is the fraction of pure states for which the variable is forced to unsatisfy 
the constraint. Note that (5m-s.cz(0, 0) = 1 — Qfu^cz ~ Qfu^cz- Let us now 
parametrize the messages from constraints to variables. If variable Xiu enters 
unnegated in constraint cz, then certainly constraint cz does not force it to 
take the value 0. Thus Qcz^iu{0, 1) = 0, and the message can be parametrized 
by the single number Qcz^iui^, 0). On the other hand, if variable Xiu enters 
negated in constraint cz, then certainly constraint cz does not force it to 
take the value 1. Thus Qc2->m(l,0] = 0, and again the message can be 
parametrized by the single number Qcz^m(0, 1). We set 



Qc 



j(0, 1) if Xiu is negated in cz, 
j(l, 0) if Xiu is not negated in cz. 



Message Qcz^iu is the fraction of pure states for which cz warns iu to sat- 
isfy it. The survey propagation equations fll44p . f ll45p then become (recall 
d{bv,iu) = 1 (resp. 0) for a dashed (resp. full) edge {bv,iu)), 

Qcz^iu = n ^ 

(17) 

jv£d{cz)\iu 
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and 



bvGd{iu)\cz 



Y\_ (-'- ^ Qbv^iu) 

bv£d{iu)\cz 



Ql 



u ~ 

iu^cz 



*^(bv,iu) ^(iu.cz) ^ ^{bv ,iu)¥^^{iu,cz) 

W (l-Q,™)Wl- W {l-Qbv^ 

bv£d{iu)\cz bv£d{iu)\cz 



where = means that the r.h.s has to be normahzed to one. Define 



{bv 



■^lu-^cz ■ n i'^- Qb 

bv£d(iu)\cz 

Qiu^cz — Y\ ^ Qbv^iu)- 

bv£d{iu)\cz 

Then using (|T7I) and the normahzed form of (fT9|) 

Qjv-^czO- ~ Qjv^cz) 



(19) 

(20) 
(21) 



Qc 



n 



jv£d{cz)\iu 



Qiv-^cz ~^ Q 



Q jv-^czQ jv-^ 



(22) 



We wiU work with the set of SP equations fl20l) . fl2Tl) . fl22|) . The complexity 
becomes 



Sl,«,(«) = 



• C2 iz (cz,iu) 



(23) 



with 



^cz lll'l (Qm-^cz"'"Qm-i>c2 Q iu^czQ iu^cz) Qm^-i-czl-'- Qiu-^cz) p 

(24) 



'inSc'(cz) 



'^(6t),iz>— 



— ln<| JJ^ (1 — Qhv-^iz) + JJ^ (1 — Qbv~>iz) — W (1 — Qbv-fiz] 
bv£d{iz) bv£d(iz) bv£d{iz) 



In (Q 



(25) 
(26) 
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The set of SP equations fl2Dl) . (^T^, f l2^ is solved under the following assump- 
tions. We treat the set of messages emanating from a constraint at position 
z, namely Qcz^iu for u = z, . . . , z + w — 1, as i.i.d. copies of a r.v. Qz de- 
pending only on the position z. Similarly we treat the messages emanating 
from a variable node at position u, namely Q^^cz for 2; = m — w + 1, . . . , m, 
as i.i.d. copies of a r.v. Q^. Now, fix a position z and pick p, q two in- 
dependent Poisson(^) integers. Pick ki, . . . , kp+g independently uniformly 
in {0, . . . ,w — 1}. Similarly, pick /i, . . . , Ik-i independently uniformly in 
{0, . . . ,w — 1}. Under our assumptions the SP equations becom^ 

Q^=n(i-C), (27) 

i=l 

p+q 

q:= n (28) 

i=p+l 

and 

n — TT ^z+iM ^z+ij) /^Qx 



=1 Qz+k + ^z+/, Qz+uQ 



The boundary conditions can be taken into account by setting Qz = ^ for 
z < z > ^. These equations are solved by the standard method of 
population dynamics. It is then possible to compute the average complexity 
from 



El,.(«) = ^ (aE[Sri+E[Sr]-«i^E[Sf^^]), (30) 

where 

= in(n(^?ai + QiS. - QlSs^'S:) - n^^S!(i - q-zS.)]^ m 



1=1 i=l 

p p+q p+q 



in{n(i - C) + n (1 - - 11(1 - c J}, (32) 

i i=p+l i=l 



(33) 



Figure [XT] shows the average complexity for the regime N » L » w, 
for K = 3 and w = 3. We find it is positive in an interval [<ysp,w,L, l] that 



^In (P7)) . (pS)) . equalities mean that the r.v. have the same distribution. 
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^L.w{a) 
A 

0.02 



0.01 - 




Figure 3: Average complexity versus a for tire [1000, 3, a, 3, L] ensembles with 
L = 10 (rightmost curve), 20, 40, 80 (leftmost curve). Values of the corresponding 
thresholds are given in table [TJ 



individual 


asp 


as 


L = 1 


3.927 


4.267 


coupled 


ttSP,L,3 




L = 10 


4.386 


4.663 


L = 20 


4.274 


4.425 


L = 40 


4.269 


4.335 


L = 80 


4.268 


4.301 


L = 160 


4.267 


4.284 



Table 1: SP and static phase transition thresholds of the [1000, 3, a, 3, L] ensem- 
bles. 



shrinks down to zero as L increases. The SP and phase transition thresholds 
corresponding to Figure 13711 are given in Table [1] We observe that as,w,L i «s 
as L increases. Moreover we observe that the SP threshold saturates, namely 
«SP,«),L — ^ «s for L » w » 1. 

For moderate values of L we have < asp,w,L- However since asp,w,L < 
as,w,L and lim2,_>>+oo = «s, for L large enough and fixed w we necessarily 

have asp,w,L < cts- This turns out to be difficult to observe within population 
dynamics experiments, but can be checked in the large K limit. 

3.2 Survey propagation for Large K 

For large K one can derive approximations of the survey propagation equa- 
tions that lend themselves to more explicit analysis [39] . We will not attempt 
to control the error terms, but it is known for the individual system that the 
approximations are excellent already for K > 5. We can check numericaly 
that this is also the case for the coupled-CSP. 

Fixed point equations. Following [52], we introduce entropic random 
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variables 

g, = -ln(l-4), gf = -lnQ±. (34) 
From ([27D, (EHD and (EHD we obtain 

p p+q 



and 



we set 



i=l i=p+l 



'z + li 



E[g±] = xf and E[g,] = y,, (37) 

for the averages over the graph ensemble. The number of i.i.d. random 
variables in fl35|) is a Poisson(^) integer. Therefore we assume that for 
large K the r.v. qf are self-averaging. It is reasonable to expect that they 
can be replaced by their expectation in fl5H]l and that hence is also self- 
averaging. This implies a closed set of equations for the expected values of 
messages, 



•^z ~ 2w l^k=0 yz-k, 



(3^ 



We further approximate (1381) . A self-consistent check with the final solu- 
tion shows that = 0{K) and hence the product in the log is 0(2"^). 
Linearizing the logarithm yields 

w-l _ K-1 x~, r 1 "^"1 x~ , 1 \ K-l 

El TT e - 1 I 1 v-^ e^+fe - 1 I , , 

fci,...,fcK-i=o i=i 2e - 1 1^ k=o '+1^ - 1 ) 

It is convenient to introduce the rescaled parameters 

a = 2-^«, = 2^-i«i^y,. (40) 

From (l34l) we see (/^^ is a measure of the average (over the graph ensemble) 
probability (over pure states) that constraints at position z send warning 
messages. From now on we write x^ instead of xf. The fixed point equations 
become 



^ 1 \-^w—l 



z—ky 

~, ^ ) 1 v^ui— 1 e^z+i—i 
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(41) 



Hence, the profile {^Pz} satisfies 

<^.^aA^ -5^^^— 3T- ^ . (42) 

These equations have to be supplemented with the boundary condition ipz = 
for z < -| and z > ^. 

The average complexity. Let us now express the complexity in terms of 
the fixed point profile. Let us first compute the contributions of variable and 
constraint nodes, and of edges. 



Contribution of variable nodes. From f l32|) . flMj) and ([3] 

E^""" = ln{e"^^=i'^"-'=» + e'^^'-p+^^'-^i - e"^^=i }. (43) 

For K large the sums in the exponentials concentrate on their averages, so 
that 

Epr] ~ ln{2e-^^^-"o _ g-^ }. (44) 

Contribution of check nodes. From fl3Tl) . flM|) and fl37j) 



't=i 

i=i - 

w~l , ^ K K 



W~L if 1 

Zi Ik=0^ M=1 i=l 

(45) 

Factoring the first product out of the log we get 

1=0 h,...,iK=o ^ i=i ^ e. ) 

(46) 

Since the ratio in the second log is 0(2 ^) we can linearize and obtain 
E[sr"1 ~ — V ln|2e-"^~+' - 6-'"-"+' ) - <^ - V ~ I . (47) 
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Contribution of edges. Similarly from fl5^ . (^^, (I57|) we have 



^ w—l 
^ /=0 



1 "'^^ r 

/=0 



Now, using f l5S]) we can express the total average complexity ( 13 Up in terms 
of rescaled variables (HQl). We find 



1 

z=-#+l 



(49) 



with 

f ^ w-l T ^ K 

(y&,w,L{z) ^ ln{2e" ^fe=o - e" » ^^=0 '^^-^ } - 2^a< — 2^ 



w ^ 2e^-+' - 1 



2^ InU - — -{I - e"^^5^ 



w 



2e^-+i - 1 



(50) 



Within our approximations the third term can be simplified further because 
1 _ g-lfi^ = 0(2-^) and we may linearize the log. Thus the second line 
in fl50|) can be replaced by 



1 I e^=+' — 1 



w ^ 2e^'-+' - 1 

1=0 



(51) 



The complexity fl49p can be viewed as a functional of the profiles {xz^^z} 
with boundary condition (/j^ = for z < — |^ and 2; > One can check that 
the stationary points of this functional are given by the fixed point equations 
(SB. 



3.3 Solutions for Large K 

We use the notation f = g to mean that lim/^f^+oo ^ = 1. The large K 
results for the individual system |39] are recovered by setting L = = 1, in 
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which case the fixed point equations reduces to 

09 aK{ Y 

One may easily check that this is the stationary point equation for the com- 
plexity (gHD function of (and a fixed), 

E,,(d, ^) = ln{2e- - e^-} - Si^Aj^^}"^ + V^{^^}- (53) 

Thus, fixed points of (l52l) are stationary points of (153|) : stable fixed points 
correspond to minima and unstable ones to maxima. 

The curve a{(p) is shown as the dotted curve in Figured This function 
is convex and has a unique minimum at <y9sp = In(ii^lni^) and d((/?sp) = 
dgp = i^l^. Near this minimum we have «((/?) ~ (^^^p")^? 7sp = thf^f • For 
(y9 >> y^sp we have Q;(</9) = -^(v^ — </2gp) and for < (/? << (/?sp we have 
q;((/?) = ^. Therefore the trivial fixed point = is unique for a < dsp, and 
there are two extra non-trivial fixed points for a > asp- Only one of them is 
stable and forms the branch (y^mst ~ Ka + ipsp for >> ipsp- 

For a < dtspi the function (!53l) has a unique minimum at (y9 = 0. For 
a > otsp a second metastable minimum appears at i^rast ~ Ka + ifo,^. At this 
minimum we find Ei i(q;, (pmst) = ln2 — d which counts the number of clusters 
as long as it is positive. Summarizing, the complexity vanishes for a < Q-sp-, 
and equals (In 2 — a) for a G [d5P,ln2]. In particular the static phase 
transition threshold is = In 2. Beyond the static phase transition threshold 
the complexity is negative and looses its meaning (one has to modify the SP 
formalism used here). Higher order corrections can be computed in powers 
of 2"^, see [39]. 

Let us now discuss the coupled case. The picture which emerges is similar 
to the one for the much simpler Curie- Weiss Chain model [5] and coupled 
LDPC codes over the binary erasure channel [2]. Before discussing the nu- 
merical results we wish to give a heuristic argument that "explains" why 
threshold saturation occurs. The argument can presumably be turned into a 
rigorous proof using the methods in [2] for LDPC codes on the binary erasure 
channel, or more general methods developed in [IQ]. 

For the sake of the argument suppose that we fix d > dsp and that 
we look for profile solutions of fH21) . on an infinite chain L — )■ +oo, that 
interpolate between the (asymmetric) boundary conditions (/^^ = 0, z — > — oo 
and — ^ V^mst, z +oo. We take as an ansatz, a kink approaching its 
asymptotic values (at the two ends) fast enough, with a transition region 
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localized in a region of size 0{w) centered at a position Z]^ink = {\^\ < 1/2). 
Figure H] gives an illustrative picture of the kink profile. We have 

1 - ^ ^ 1 ]^ 

^=L ^ ^""^ L^2 " ^^^^ 

Also, it is easy to see that the associated complexity as a function of ^, or 
equivalently is approximately given by a convex combination of the two 
minima of ^p) (given in fl53|) ) which correspond to the two points ^9 = 

(with S = 0) and = ipst (with S ^ ln2 — d). More precisely, 

^k^nk{0 ~ ^ [(^ + X + (^ - ^L) X (In 2 - a)] 
« ^(ln2-d). 

^mst 



= (S = 0) 
-\ 




ifi = V'mst (S w In 2 - a) 



L I 1 Zkink=^L L 

2 ^ 2 



Figure 4: An illustrative picture of a kink-like ansatz {y?^} ^_ ^^.^ for a solution of 

()42p . At the right end, the kink converges to the value ip = ipst (with corresponding 
complexity S « ln2 — d) and at the left end it converges to (^9 = (with E = 0). 
The transition region of size 0(1/7) which is centered at z — -^kink* 



When a < cxs, the minimum is at ^ = | (^ = 0). This means that 
the kink center will form a traveling wave through the chain, and reach its 
unique stable location at the right end. On the other hand when a > ctg the 
minimum is at ^ = — ^ (^ = (prast) and the kink will travel towards the left 
to reach its stable location. Within the present approximation, for a = as 
any position along the chain is stable for the kink center. 

Summarizing, this heuristic argument suggests that for a < as the fixed 
point equations only have the trivial solution {ipz = 0}, while for a > as 
the only solution is {(pz = V'mst}- This means that the SP threshold coincides 
with as- Here, ^ has been treated as a continuous variable, which is expected 
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to be valid only in a limit of large w. For large but finite w there will subsist 
a small gap between the SP and static thresholds, and for a fixed in this gap 
only a discrete set of positions for the kink are stable. The number of such 
stable positions is roughly equal to 2L. 

We have solved (142|) numerically with symmetric boundary conditions 

L 

'^z = 0, z < z > ^ and fixed Tp = tJ2^-_l'^z- In order to find a 
solution for all values of (p we have to let a vary slightly. In other words we 
find a solution {a(Tp); {(Pzijp)}) that is parametrized by Tp. Define the van der 
Waals curve (Figure |5]) as the function q;(^). The minimum of the van der 
Waals curve yields (as for the individual system) the SP threshold asp,u;,L 
(see Table [2] for numerical values). 




































































































































































































































2.5 







2.5 



Figure 5: Left: sequence of van der Waals curves a(jp), for K = 5, w = 3 and 
L = 10,20,40,80 (top to bottom). For Tp G [(/?mst5+c«] they converge to the 
individual system curve. Right: a magnification of the plateau region for K = 5, 
w = 3 and L = 40 shows the fine structure. The dotted line is the curve for 
the individual system and the red line shows the static phase transition threshold 
do = 0.666. 



As L increases, the curves develop a plateau at height ^ for the interval 
^ e [0,(/9mst]- Moreover they converge to the van der Waals curve of the 
individual system for Ip e [ipmst, +oo[, a fact that is consistent with theorems 
[T], [2J Precise enough numerics show that as long as w is finite the curves 
display a fine structure in the plateau interval: the magnification in Figure E] 
shows wiggles of very small amplitude. We observe that their amplitude 
decays as w grows and K is fixed (we expect from [5] that this decay is 
exponential); and grows larger as K increases with w fixed (see Table [2]). 
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-iim.. -.11111- iiiiii. .111111 

+ l k -L,i k 

2^ 22^ 2 

Figure 6: van der Waals curve in the wiggle region for the coupled system (top) 
for K = 7, w = 4 and L = 40. The red line is at the static phase transition 
threshold. The left point {Tpi,ai) = (1.657,0.678274) corresponds to the warning 
(middle left) and complexity (bottom left) profiles. In the latter the height of the 
middle part is as - ai ^ 0.010. The right point ((^^,0^) = (3.00585,0.688847) 
corresponds to the warning and complexity profiles on the right. 

Figure [6] illustrates the solutions of the fixed point equations for a in 
the wiggle region for large K. The top curve is the van der Waals curve in 
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K 


5 


7 


10 




0.666 


0.686 


0.692 


asp 


0.513 


0.449 


0.370 


«SP,80,3 


0.672 


0.682 


0.651 


"SP,80,5 


0.672 


0.688 


0.691 


Q^SP,80,7 


0.672 


0.688 


0.692 



Table 2: SP Thresholds of the individual {L = w = \) and coupled ensembles 
(L = 80, w = 3, 5, 7) are found from the van der Waals curves. For X = 10 we 
clearly see that the SP threshold saturates to from below as w increases. 

the wiggle region. The middle left warning density profile is the fixed point 
solution corresponding the left point with coordinates {Tpi.ai). Note that 
Oil = otsp,L,w For this point the total average complexity is approximately 
equal to ^^(ds — d;). The bottom left curve shows the complexity profile. In 
the middle part, the height of this profile is approximately (d^ — d;) ~ 0.010. 
Consider now the point on the right with coordinates ((^^, d,.). Note that we 
take this point very close to the static phase transition threshold d,. ~ d^. 
As a consequence the total average complexity nearly vanishes. The middle 
right warning density profile is fiat over the whole chain, except near the 
ends because we enforce the boundary conditions, and the complexity density 
nearly vanishes except in the transition regions. 



4 Coupled Q-Coloring Problem 
4.1 Numerical implementation 

First we introduce an adequate parametrization of the messages (see e.g [TT]). 
The warning vectors {Ej^^cziX)^ ■ ■ ■ , Ej^^^ziQ)) fall in two categories: those 
that have exactly one zero component; and those that have at least two zero 
components. For coloring, equation fll37p becomes 

Ecz^iui^iu) — niin |l(xc2 = Xj^^ + Ejy^c^ixjy)^ Ccz^iu- (55) 

It is easy to see that when Ejy^cz has exactly one zero component, then 
Ecz^iu has exactly one non-zero component. On the other hand, when Ej^^cz 
has at least two zero components then all components of Ecz^iu are zero. 
Hence, the vector {Ecz^iui^), ■ ■ ■ , Ecz^iuiQ)) can take only Q + 1 possible 
values which are the (0, ■ ■ ■ , 0) = * vector and the Q canonical basis vectors 
(1, 0, ■ ■ ■ , 0) = 1, (0, 1, 0, ■ ■ ■ , 0) = 2, (0, ■ ■ ■ , 1) = Q. The interpretation 
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is clear: a warning vector G {1, ■ ■ ■ ,Q} forces the variable to choose a color, 
while a warning vector * leaves the variable free. 

We can rewrite the SP equations in terms of the distribution of warnings 
Qcz^iu{o,), 0, G {1, ■ ■ ■ ,Q,*}. Since constraints have degree 2 we can view 
the messages Qcz-5.m(a) as carried by the edge {jv, iu), where jv is the unique 
node in d{cz) \ iu. We thus make the replacement Qcz^iu{ci) Qjv^iu{o) 
and write down the SP equations on the induced graph of variable nodes. 
Moreover following [H] we seek solutions that do not depend on colors, and 
set Qjv-^iu{,o) = Qjv^iu for a G {1, . . . ,Q}. A calculation then shows that 
(dm), ffTlH|l reduce to 

Q _ J2?=li~^y I )YlkwGd{jv)\iui^ ~ ^ ^)Qkw^jv) ^^^^ 

J2b=0 (i+l) Y[kwGd{jv)\iu(^ ~ + ^)Qkw^jv) 

and 

Now, recall that the degrees of nodes of the induced graph are Poisson(2a) 
integers. From now on we set c = 2a. We solve (jSSD, dSZD under the as- 
sumption that the messages emanating from node jv are i.i.d. copies of a 
random variable with a distribution that depends only on the position 
V. Fix a position z, pick an integer d according to a Poisson(c), and pick 
integers fci, ■ ■ ■ , /c^ i-i-d. uniform in {—w + 1, ■ ■ ■ ,w — 1}. We hav^ 

Ea-.'(-i)'G?i)nti(i-(' + i)«it.)' 

and 

Q.{*) = l-QQz. (59) 

Here, the relevant boundary conditions are taken into account by setting 
Qz = for z < —-^ and z > These equations are solved numerically by 
population dynamics. This then allows to compute the ensemble average of 
the complexity, 

^lAc) = ^ E (Epr]-^E[Ef^^]), (60) 

z=-^+l 



^Interpreted as equalities between random variables. 
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Q 


3 


3 


4 


4 


individual 


CsP 


Cs 


csp 


Cs 


L = 1 


4.42 


4.69 


8.27 


8.90 


coupled 


CSP,L,3 


Cs,L,3 


CSP,L,3 


Cs,L,3 


L = 10 


4.874 


4.879 


9.20 


9.25 


L = 20 


4.70 


4.75 


8.91 


9.06 


L = 40 


4.69 


4.72 


8.91 


8.98 


L = 80 


4.69 


4.70 


8.91 


8.93 



Table 3: Thresholds computed by population dynamics for the individual and 
coupled ensembles for ensembles [1000, Q, c, 3, L]. 



with 




Figure 7: Average complexity for the [1000, Q, c, 3, L] ensembles with Q = 3 (left) 
and Q = 4 (right). Here L = 10, 20, 40, 80 from right to left. See corresponding 
table O for numerical values of thresholds. 

The numerical results are similar to those for the coupled i^-SAT model. 
Figure [7] shows that the complexity is positive in an interval [csp,w,l,Cs,w,l] 
which signals the existence of exponentially many pure states. Beyond Cs^w,l 
the complexity becomes negative which means that the graph instances are 
not colorable w.h.p. Table [3] gives the values of these thresholds. Again, we 
observe that Cs^w,l i Cs as L increases, and that threshold saturation takes 
place, namely csp,w,l Cs a.s L » w » 1. 
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4.2 Survey propagation for large Q 

Fixed point equations. The large Q analysis for the individual system [H] 
is extended to the coupled model. In this regime Cg is 0{Q\yiQ) therefore it 
is natural to set 

c = eg In g, (63) 

and to analyze fl58l) for c fixed. In this limit the node degrees concentrate on 
d ^ QhiQ with a fluctuation 0{y/Q In Q). Therefore, we assume that in the 
expression 

n(i-('+i)<3i!i..)=e^- (64) 

i=l 

we can replace the sum over a large number of terms by its average, 

cQ 111 Q — 1 



e 2m- 



In this expression the average over d and ki, . . . ,kd has been carried out and 
the remaining expectation is over Q^. Since the product flM|) enters in fl58ll . 
we conclude that Qz concentrates on its average. Thus, setting 

E[g,] = (66) 

for the average warning probability, we find 



(67) 



It can be checked self-consistently from the solutions of the fixed point equa- 
tion that (jz = 0(g~^) and therefore for / = 0(1) the log in (p71) can be 
linearized, while the terms with higher / are damped. Linearizing the log the 
sum over / can be performed, and working with rescaled variables 

= (cging)g„ (68) 

we find 

0. ^ aging^|f^^'(;CI^"!"#^^-"'^"^l- m 

Let 



27 



The fixed point equation takes the simple form 



^ w—l 



(71) 



k=—w+l 



These equations must be solved with the boundary condition 6*^ = for 
z < and z > ^ in order to find the average warning probability profiles. 

Average complexity. Proceeding as above we find from flMl) . fl^2|) 

E[En=ln||^(-l)'(^^^)e^S^"-"-'^+ii"^i-^ (72) 

^ w—l 

Epri = 7^ 7 ln{l-Qg,g,+,}. (73) 



2w 



k=—w+l 



As before since Qz = 0{Q~^) we can linearize the log in the exponential 
of the first equation and the one in the second equation. Then working 
with the rescaled variables (HUj) . straightforward algebra leads to an average 
complexity fl60l) given by 



1 11 
Si^^(c) = - in|l - (1 - e~^^fe=-»+i^-+^)^} 



+ 



2clng2w; 



fc = — UI + l 



(74) 



This functional is defined for profiles that satisfy the boundary condition 
Oz = for z < and z > ^. The consistency of our approximations can 
be checked by noticing that the stationary points of (174|) are precisely given 
by the solutions of the fixed point equation ( ITB . 



4.3 Solutions for large Q 

The discussion is quite similar to the case of i^-SAT so we will be brief. 
By setting L = w = 1, we recover the fixed point equation of the individ- 
ual system which is 6* = cFq{9). Fixed points are stationary points of the 
complexity function of 6, 

E,,(c,^)=ln{l-(l-e-r} + :5^ (75) 
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This function controls the existence and nature of the fixed points. At 9 = 
it has a minimum for all c which corresponds to a trivial stable fixed point 
and a vanishing complexity Ei^i(c) = 0. It is unique for c < csp = 1. For 
c > csp a. second minimum appears. This corresponds to a stable fixed point 
solution which form the branch ^mst ~ clnQ + ln{QlnQ). Replacing in (175|) 
we find Si^i(c) (1 — |)lnQ. This is positive in the interval c G [1,2], 



and looses its meaning beyond Cg 
threshold. 
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2 which is the static phase transition 
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Figure 8: Left: sequence of van der Waals curves for if = 5, w = 3 and L = 
10,20,40,80 (top to bottom). They converge to the individual system curve for 
G [^mst ) +oo] . Right: a magnification of the plateau region for L = 40 shows the 
fine structure. The dotted curve is the individual system curve and the red line 
corresponds to the phase transition threshold cng = 1.840980. 



Let us now turn to the coupled model. The same heuristic arguments 
than for ii'-SAT hold. In the regime L » w » 1 and with asymmetric 
boundary conditions 6*^ — )■ for 2; — t- —00 and 6 — )■ O^^st for z — +00 we take 
a kink-like ansatz for the solutions of fl7T]) . Their total average complexity 
is given by a convex combination of, Si_i(c, 6* = 0) = and Si i(c, 6'mst) = 
(1 — |) InQ, with weights determined by the location of the kink center ^L. 

We have 6 = ^t-^+i ^z = {\- O^mst and S^.^^ ^ £^(1 - f ) InQ. The 

stable kink fixed point profile corresponds to ^ = and {9^ = 0} for all 
c < 2 which means that the complexity vanishes for c < 2. Thus within 
this approximation the SP threshold saturates to the static phase transition 
threshold. 

We solve (j7T|) numerically with symmetric boundary conditions which 
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Figure 9: van der Waals curves of the coupled system (top) for Q = 7 , w = 2 
and L = 40. Middle and bottom left are the warning and complexity profiles 
corresponding to the point with coordinates (Ouci) = (1.30125,1.91260). Notice 
that Q is near the phase transition threshold so that the complexity nearly vanishes 
except at the transition regions of the kink. The total average complexity nearly 
vanishes. Middle and bottom right are the warning and complexity density profiles 
corresponding to the point {9r,Cr) = (2.165, 1.906734). For the complexity profile 
at the bottom right, we see that in the middle region of the profile the height of 
the complexity density is (1 — ^) w 0.047 and the total average complexity is 

approximately given by ^^(1 — ^)ln(5. 
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Q 


5 


7 


10 


Cs 


1.840980 


1.911260 


1.635790 


CSP 


1.6411666 


1.651565 


1.949869 


CSP,80,2 


1.839709 


1.906734 


1.939527 


CSP,80,3 


1.840978 


1.911213 


1.949606 


CSP,80,4 


1.840980 


1.911260 


1.949865 



Table 4: SP Thresholds of the individual {L = w = 1) and coupled ensembles 
(L = 80, w = 2,3, 4) are found from the van der Waals curves. Threshold values 
Cs are from population dynamics. For Q = 10 we clearly see that the SP threshold 
saturates to Cs from below as w increases. 

enforce the profile to vanish at the end points of the chain. There exists a 
family of solution profiles {c{6), {6z{6)}) parametrized by the total average 

warning probability 9 = ^z- Figure [S] illustrates a sequence of 

van der Waals curves c{6) and Table H] gives numerical values of their minima 
which determines csp,ui,l- 

Finally, Figure [9] displays warning and complexity profiles for c in the 
wiggles region. The results are analogous to those of Ji'-SAT. 

5 Proofs of theorems [T] and [2] 

In this section we sketch the proofs of theorems [T] and [2j The proof of theorem 
[T]is straightforward and does not depend on the details of the model at hand. 
On the other hand that of theorem [2] has to adapted for each model at hand. 

5.1 Proof of theorem [1] 

Recall that for the Hamiltonian of the open chain T-Lcouix) in (|3]), x = (xiz) 
with {i,z) G U^^_L_^]^ ... L.^^„]^\4. It will be convenient to set x = (x',a;") 

where x' = (xj^; z = + 1, . . . , -1) and x" = [xiz] z = |^ + 1, . . . , + w — 1). 
Recall also that the Hamiltonian l-C^^ni^) of ^^e periodic chain is given by 
the same expression ^ with x' = (xj^; z = ^ + 1, . . . , Therefore the 
difference between the two Hamiltonians only comes because of the terms 
i^cz{x_Q^^) with z=-|— w + 2, In other words, 

l^coufe', x") - I < Mw, (76) 

for all x". Now, from the fact that 

niZ.{^d) -Mw< -Hcouix, x") < n^.Zi^) + Mw, (77) 
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and by taking the min, dividing by NL and taking the expectation, we deduce 

QIW ctw 
e^^,L,«,(«) ~ X - - ^N!L,wi(^) + (78) 

which proves the theorem. 

5.2 Proof of Theorem [2] 

As explained in section |5] the proof that the hmit exists, is continuous and 
non-decreasing, for the individual models is provided in [18] and is essentially 
the same for the coupled periodic chain. Here we prove the equality of the two 
limits (fTOj) . The following notation is convenient. For a given graph instance 
G (from some ensemble) we call 'Hg(^) the corresponding Hamiltonian. It 
always consists, as in (E]), of a sum of terms 1 — ipczi^Qcz) over constraints 
{c,z) G G . The ground state energy is equal to mm^T-Lcix). To set up 
suitable interpolation procedures, it is convenient to first define three extra 
ensembles. 

The "connected" ensemble. This is essentially the individual [N, K, a] en- 
semble scaled by L. We have a set of LN variable nodes and a set of LM 
constraint nodes. Each constraint node has K edges connected u.a.r. to 
variable nodes. Expectations with respect to this ensemble are denoted by 
Econn- Because of the existence of the limit we have 

lim -^Econn[min'HG(^)] = hm Cat (a), (79) 

for any fixed L. 

The "disconnected" ensemble. This is a variant of the individual [A^, K, a] 
ensemble replicated L times. We place at positions z = —^ + 1,...,^, 
L disjoint sets of variable nodes containing each nodes. Each node 
from the set of LM constraint nodes is affected u.a.r. to a position z = 
— "I + 1, . . . , ^. Note that the set of constraint nodes at position z has 
cardinality ~ Bi(LM, j^). Each node from Cz has K edges that are 
connected u.a.r. to nodes in Vz. Expectations are denoted by Ejisc- Since 
each Mz is concentrated on M with a fluctuation 0(a/M), we can show by 
an argument similar to the proof of theorem [1] that 

-4Edisc[min?^G(ai)] = e;v(«) + 0{N-'/^), (80) 

where 0{N~^^'^) is uniform in L. 
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The "ring" ensemble. This is a variant of the periodic chain in section [2J 
We place at positions z = + 1, . . . , ^, L disjoint sets of variable nodes 
Vz, each containing nodes. Now we have a set of LM constraint nodes. 
Each constraint node is affected to a position z = + 1, . . . , ^ u.a.r. and 
(say the position is z) its K edges are connected u.a.r. to the set of variables 
^k=o^z+k mod L- Note that the sets of constraint nodes have cardinalities 
Mz ~ Bi(LM, j^). We denote by Ej-ing the expectation with respect to this 
ensemble. Since each is concentrated on M with a fluctuation 0(a/M), 
an argument similar to the proof of theorem [1] shows that 

-^EHng[mm?^G(x)] = el'^Ja) + 0{N-'/'), (81) 

where 0(A^~^/^) is uniform in L (and depends on w). 
We will show 

Econn[min'HG(^)] < EHng[min'HG(^)] < Edisc[min'HG(^)], (82) 

XXX 

which allows to conclude the proof of the theorem by using (17^ . fISD]) . (|5T]) . 

Left inequality in (|5^ . We build a sequence of interpolating "r-ensembles" , 
r = 0, . . . , LM, interpolating between the ring (r = 0) and connected (r = 
LM) ensembles. We have two sets of LM constraint and LN variable nodes. 
The variable nodes are organized into L disjoint sets Vz each containing N 
nodes, placed along the positions z = —^ + 1,...,^. Expectation with 
respect to the r-ensemble is denoted by E^,. To sample a graph Gr from this 
ensemble we first take r nodes - called type 1 - from the set of LM constraint 
nodes. Each one has K edges which are connected u.a.r. to the set of LN 
variable nodes. For the remaining LM — r constraint nodes - called type 2 - 
we proceed as follows: each one is affected u.a.r. to a position z, and its K 
edges are then connected u.a.r. to the wN variable nodes in U^Jq V^+a.- mod l- 
We claim that for 1 < r < LM, 

Er [min (x)] < E,_i [min nG,._, (x)] . (83) 

X X 

Clearly this implies the left inequality in fl82|) . Let us prove this claim. Take 
a random graph Gj. and delete u.a.r. a constraint from the type 1 nodes: 
this yields an intermediate graph G. One can go back to a random graph Gr 
by adding back a type 1 node according to the above rules, or one can go to 
a random graph Gr-i by adding back a type 2 node according to the above 
rules. We will prove that conditioned on any realization of G we have 

E, [min Hg^ (x) \ G] < E,„i [min Hg^_, (x) \ G] . (84) 
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Claim f lS^ follows by averaging over G. We now prove flMl) for K-SAT and 
Q-coloring separately. 

K-SAT : Consider the set of "optimal assignments" x that minimize 'Hq{x). 
We say that a variable is frozen iff it takes the same value for all optimal 
assignments. We call J-" the set of variable nodes with frozen variables and 
J-'z = J-Tl Vz- Now consider adding a new constraint node n to the graph G. 
This will cost an extra energy iff the node n connects only to frozen variable 
nodes and does not satisfy them. For such an event we have 

mm'H.Qu Jx) — min'HMx) = 1. (85) 

X X 

When the node n is connected u.a.r. to the LN variable nodes {n is type 1) 
this event has probability -^i^)^ ■ Thus 

Kimm-HcM I G] - mm?{^(x) = ^ (^^^ . (86) 

Similarly when the node n is affected u.a.r. to a position z and then connected 
u.a.r. to U^jJVz+fc mod l {n is type 2) we get 



1 X 1.1"-' 



E,,_i[minHG._,(x) | G] - YmnUaix) = j ^ ^(tT^ XI l-^^+fcmodL|)^- 



-4+1 *^=o 



2 



^7) 



Claim f lM|) follows from the last two equations, convexity of x^ for a: > 0, 
and 

2 ^ W~\ 

\^\= / , z+k mod l|- (88) 

Q-Coloring: The proof is similar. Consider the set of "optimal colorings" 
that minimize 'Hq{x). We define an equivalence relation between variable 
nodes: we say that two nodes are equivalent iff their two colors are identical 
for all optimal assignments. Let J^-', 1 < j < J, be the equivalence classes of 
nodes and let J-'f = J^^CiVz- Now, assume we add a random constraint node n 
to G. We have min^ "H^^^ (x) — min^?/g(x) = 1 only when n chooses its two 
variables from the same equivalence class; otherwise the energy difference is 
zero. Thus we obtain 

E, [mm Hg^ (x) \ G] - mm Ha{x) = (j]}) ' (§9) 
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and 



E,_i[mm'HG._i(^) | G] -mmT^cl^) = i modLl)^- 

(90) 

Claim (184|) then follows from the last two equations and convexity of x^. 

Right inequality in (l82ll . We construct new r-ensembles, r = 0, . . . , LM that 
now interpolate between the disconnected (r = 0) and the ring (r = LM) 
ensembles. A random graph Gr is constructed as follows. We have a set 
of LM constraint nodes and a set of LN variable nodes organized into L 
disjoint sets each containing nodes, placed along positions z. We first 
take r constraint nodes, called type 1. Each of them is affected u.a.r. to 
a position z, and its K edges are connected u.a.r. to variable nodes in Vz. 
Next, the remaining LM — r constraints nodes - called type 2 - are each 
affected u.a.r. to a position z and its K edges are connected u.a.r. to wN 
nodes in U'^IgV^+k mod l- Note that at each position there are Bi(r, j^) type 
1 nodes and Bi (LM — r, j^) type 2 nodes, so in total there are Bi (LM, -^) 
constraint nodes. Similarly to the previous interpolation we will prove 

ErlminnGM] <E,_i[min-HG..i(x)]. (91) 

X X 

This inequality implies the upper bound in f E2]) . To prove f lHT]) . as before, we 
consider the random graph G obtained by deleting u.a.r. a type 1 node from 
Gr- From G one gets a random graph Gr by adding back a type 1 node, or 
one gets a graph Gr-i by adding back a type 2 node instead. We first prove 
that 

E, [min -Hg. (x) \ G] < E,„i [min 7^g._i (x) \ G] , (92) 

X X 

and then by averaging over graphs G we get ( 19T]) . Let us briefly sketch the 
derivation of (192 p . 

-ft"-SAT : We use the same sets J^z of frozen variables at position z corre- 
sponding to the ground state configurations of 'Hq{x). We have 

ErlrmnncM \ G] - rmn-Haix) = ^ ^ ^(^) ' ^^^^ 
- - .=-1+1 ^ ^ 



35 



and 



2 -, W — 1 



1 1 ^ ^ 

E,_i[min7/G,_i(^) I G] - minHG(x) = y XI XI l-^-+femodL|)^. 



(94) 



Estimate fl92|) then follows by the convexity of the function for x > 0. 
Q-coloring: We use the same equivalence relation between variable nodes and 



sets Fl- We have 



~ J 2 

EriminHcM I G] - mmUaix) = y E E(^) ' ^^^^ 



and 



i 1 1 i=i 

2 



2 J W — 1 



K^i[mmncUx) I G] - muin^ix) = y ^1 E l-^+.modLl)'- 

z=-^ + l i=l k=0 

(96) 

Again, estimate f l92|) then follows from the convexity of x"^. 



6 Dynamical and condensation thresholds 

The SP formalism says nothing about the relative sizes (internal entropy) of 
clusters of solutions and consequently does not take into account which of 
them are "relevant" to the uniform measure over zero energy solutions. For 
similar reasons, it is not clear that the SP threshold has particular algorith- 
mic significance. These issues are partly addressed by the more elaborate 
entropic cavity method [19], [20], [21]. It predicts the existence of the dy- 
namical and condensation thresholds and Oc- The dynamical threshold 
is believed to separate a phase [a < aa) where the uniform measure is es- 
sentially supported on one well connected cluster of dominant entropy, and 
a phase {ad < a < ac) where the measure is supported on an exponential 
number of clusters with equal internal entropy. For a > ac the measure 
condenses on a "handful" of clusters of dominant entropy. The condensation 
threshold is a static thermodynamic transition in the sense that the total 
ground state entropy has a non-analyticity as a function of a. The algorith- 
mic significance of ad and/or ac is still unclear. But see [22], [23] for recent 
related results. 
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K 


asp 


tt5P,80,3 


Old 


"d,80,3 




Q^c,80,3 




Q^s,80,3 


3 


3.927 


4.268 


3.86 


3.86 


3.86 


3.86 


4.267 


4.268 


4 


8.30 


9.94 


9.38 


9.55 


9.55 


9.56 


9.93 


10.06 



Table 5: Thresholds of individual and coupled X-SAT model for L = 80 and w = 
3. The condensation and SAT-UNSAT thresholds correspond to non analyticities 
of the entropy and ground state energy and remain unchanged (for L — )• +oo). 
Already for w = 2> the dynamical and SP thresholds saturate very close to Uc and 
as- 



We have computed the dynamical and condensation thresholds of coupled 
CSP. Let us denote them ad,L,w and ac,L,w (with w fixed). We observe that 
as L increases ac,L,w — ^ ^c- This observation is consistent with the following 
rigorous result that we prove in appendix |Xl the thermodynamic limit of the 
free energy (at finite temperature) of the chain is identical to that of the 
individual model. From the free energy one can formally obtain the entropy 
by differentiating the free energy with respect to temperature. The result 
about the free energy then suggests that the zero temperature entropy of 
the chain and individual models have the same non-analyticity points as a 
function of the constraint density. The second important observation is that 
in the regime 1 << w « L we find ad,L,w — > ^c- Thus the dynamical 
threshold saturates towards the condensation thresholc0. 

The dynamical and condensation thresholds are analogous to the dynam- 
ical and condensation temperatures of p-spin glass models for p > 3, and to 
the glassy and Kauzmann transition temperatures in structural glasses [21], 
|25j . pB] . One expects that a similar saturation of the dynamical towards 
the condensation temperature holds for coupled p-spin glass models on com- 
plete graphs for p > 3. On the other hand, for p = 2 the replica symmetry 
breaking transition is continuous, there is no dynamical temperature, and 
spatial coupling is not expected to modify the phase diagram. 

Table E] summarizes all the behaviors of the SP, SAT-UNSAT, dynami- 
cal and condensation thresholds for the i^-SAT problem. The situation for 
coloring is similar. 

^Note that for K = 3 we already have ad — etc for the individual ensemble. 
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7 An application of threshold saturation to 
algorithmic lower bounds 

We briefly discuss a methodology, that uses coupled CSP ensembles, for prov- 
ing lower bounds on the static phase transition threshold of individual CSP 
ensembles. We illustrate it with simple examples and show how threshold 
saturation can help. These examples do not reach the best known lower 
bounds, but they serve well to illustrate a new methodology for attacking 
the problem. We keep the discussion at an informal level. 

Given a CSP from an individual ensemble, one usually tries to devise 
an algorithm that provably finds solutions w.h.p for a < a^ig- This then 
imphes a^ig < «s. Consider now the coupled ensemble, and apply the same 
algorithm. Call aaig,L,w the algorithmic threshold for the existence w.h.p of 
solutions and set aaig,«; = liniL_>.+oo «aig,L,w From theorems [T] and [2] we know 
that the coupled ensemble has the same static phase transition threshold 
as the individual one, when L — )■ +00. Therefore one certainly has the 
lower bound aaig,w < The point is that for well chosen algorithms an 
improvement of the bound may occur, namely a^ig < ttaig,w < C(s, and one 
would expect to get the best lower bounds by increasing w. A well chosen 
algorithm is one that shows a "threshold improvement" or even saturation 
phenomenon. Somehow the "seed" provided by the reduced hardness near 
the boundaries should grow and propagate in the bulk. 

Below we illustrate the idea with three simple peeling algorithms applied 
to /ST-XORSAT, i^-SAT and Q-COL. 

-ft^-XORSAT . This case provides the best illustration. The individual 
model has a static phase transition at a^, and a clustering transition at asp 
with a complexity counting clusters in the interval [a^p, a^]. In appendix 
1X1 (theorem [5]) we show that the coupled and individual ensembles have the 
same phase transition threshold as for even K (for odd K the proof breaks 
down but the result is presumably true). Now consider the "leaf removal" 
algorithm. As long as there is a leaf variable node remove it, and remove the 
attached constraint node with its emanating edges. If this process ends with 
an empty graph the instance is satisfiable. It is known that this algorithm is 
equivalent to BP message passing, and the density evolution analysis leads 
to the fixed point equation 

X = (1 - exp{-aKx)^-\ (97) 

Here, x is interpreted as the probability (when the number of iterations goes 
to infinity) that a constraint node is not removed. There is a threshold air 
above which (19 7p has non-trivial fixed points (i.e, the fraction of remaining 
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K 


3 


4 


5 


7 




0.917 


0.976 


0.992 


0.999 




0.818 


0.772 


0.701 


0.595 


Q^lr, L=80, w=5 


0.917 


0.977 


0.992 


0.999 



Table 6: First line: phase transition threshold for i('-XORSAT. Second line: leaf 
removal threshold for the the uncoupled case. Third line: leaf removal threshold 
for a coupled chain with w = 5, L = 80. 



variables is positive), so we get a lower bound air < For the coupled 
ensemble the density evolution analysis yields the one-dimensional fixed point 
equations 



I w ^-^ w ^-^ I 

V ;_n I — n J 



X, 



1=0 k=0 

with boundary condition x^ = for z at the boundaries. Solving for the non- 
trivial kink solutions numerically, we indeed observe air < «ir,w < c^s- Table 
[6] shows the threshold improvement for w = 5 and the first few values of K. 
In fact the leaf removal threshold even saturates air,«, ^ f^s as w — )■ -|-oo. This 
is not surprising since for XORSAT the SP formalism leads to the same fixed 
point equations (but with a different interpretation for x and x^) [H], [32] • 
In particular there is a complexity Sxorsat(a) > for a G [a5P,as] counting 
the number of clusters of solutions in Hamming space with asp = air and 
Sxorsat(as) = 0. Note that for large K one finds air = In i^/fC + Inlni^'/i^' + 
1/K + o{l/K) and a, = 1 + o(l). 

■R^-SAT . Let us now turn to il'-SAT and consider the "pure literal rule" 
algorithm [3D], [21] • Consider variable nodes that have only one type of 
edge - dashed or full - attached to them. As long as there are such nodes 
(called "pure") set the variable to the value which satisfies all the attached 
constraints and remove these constraints and their edges. Continue until 
no "pure" node remains. If no constraint node remains then the algorithm 
succeeds in finding a satisfying assignment. This algorithm can be cast in a 
message passing form and can be analyzed by the density evolution method 
[38] . The net result is that the pure literal rule succeeds w.h.p for a < api 
such that 

a; = (1 -exp(-^x))^-^ (99) 

has a unique fixed point a; = 0. We now take coupled instances from the 
ensemble defined in section |2j In order to analyze the pure literal rule we 
can think of extending the chain to Z with "pure" variable nodes for z < 
and > ^+w. The peeling of constraints attached to pure nodes will propagate 
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inside the chain as long as a is not too large. The analysis yields the one- 
dimensional fixed point equations 

/=0 fc=0 

with boundary condition a:^ = for z at the boundaries. Note that fl99l) . 
(11 001) are the same as (197|) . fl98|) with the replacement a — )■ a/2. Therefore 
the pure literal thresholds ttpi, ftpi,^ for the individual and coupled ensembles 
are obtained just by doubling the XORSAT thresholds. For example for 
= 3 we have a^i ~ 1.636 < api,w=5,L=80 ~ 1.835 < ~ 4.26, a modest 
improvement. Interestingly when — )■ +oo we have 

2\nK , , , 

ttpi = — — — but api,w —^2 as w — j- +oo (101) 

Of course this is still a ridiculous lower bound since we know that Ug = 
2^ In 2. 

Q-COL. Finally we discuss a similar peeling algorithm for Q-COL. This 
algorithm determines the Q-core of a graph G and has been analyzed by the 
method of differential equations |33] • Here we discuss the algorithm from the 
message passing point of view. Assume there exists a node z in G that has 
degree less than Q. Clearly, if we can color the graph G \ i with Q colors, 
then G can also be colored with Q colors. Hence, finding a Q-coloring for G 
is equivalent to finding a Q-coloring for G \i. As a result, we can peel the 
node i from G and continue this process until the final graph (the Q-core) 
has no more nodes of degree less than Q. If the final graph is empty then 
the algorithm succeeds otherwise it fails. 

Now, consider the following message passing rule. At time t G {1, 2, ■ ■ ■ }, 
assign to each edge G E two messages and The messages at 

time t + 1 are evolved from the ones at time t via the following procedure: 

1. At time 0, initialize all the messages to 0. 

2. At time t + 1, 

h&di\j 

The above message passing rule is equivalent to the peeling algorithm in 
the sense that when /x*^^- = 1, the vertex i would have been peeled by the 
algorithm some time before t and if f^^^j = 0, the vertex i would not have 
been peeled by the algorithm up to time t. 
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Define Xt = P(/i*_^j = 1). we derive the density evolution equation that 
relates Xt+i to Xt- Let G be randomly chosen from G{N, -^) with very 
large. Fix an edge Observe that nl'^j = 1 if and only if the number of 

incoming messages that have value 1 is < Q — 2. Moreover, the probability 
that the degree of i is equal to (i > 1 is ^"'^■^I^YyT- Hence, we can write. 



-«-E^-'(^rT^ + E^-(^EC; (102) 



d=l ^ ' d=Q ^ ' j=0 

One can simplify this equation. Indeed, 

d=Q ^ ^' ^ j=0 ^ •' 



oo w_l d-1 X 

d=Q ^ ^ j=Q~l V / 

^ ^ j! (rf-l-j)! 

(103) 



7 

j=o 

Defining y = c(l — x) and the function 

Q-2 



G{y) = l-e-yY.'- (104) 

j=o 

we see that we have to study the solutions of the fixed point equation 

y = cG{y). (105) 

For c < Cp there is a unique trivial fixed point y = (i.e., x = 1) and the 
algorithm succeeds. Non trivial fixed points appear for c > Cp which is the 
threshold for the emergence of a Q-core. Table [7] contains the numerical 
values of Cp for several values of Q. 

We now take coupled instances from the ensemble defined in section [2l 
We can write down the density evolution equations and the corresponding 



41 



Q 


3 


4 


5 


7 


Cs 


4.69 


8.90 


13.69 


24.46 


Cp 


3.35 


5.14 


6.79 


9.87 


Cp, L=80, w=5 


3.58 


5.74 


7.84 


11.92 



Table 7: First line: static phase transition threshold for Q-COL. Second line: peel- 
ing algorithm threshold for the the uncoupled case. Third line: peehng algorithm 
threshold for a coupled chain with w = 5, L = 80. 

one-dimensional fixed point equations. Not surprisingly, similar calculations 
show that the message passing algorithm is controlled by the one- dimensional 
fixed point equation, 

^ w—l 

y. = cG(^^ E (106) 

k = --UJ + l 

where yz = c(l — x^) and Xz is the fraction of peeled nodes at position z. 
Table [7] contains the numerical values of Cp^„=5^L=80 for several values of Q, 
and shows the threshold improvement. It can be checked numerically that 
Cp = Q and Cp^u],L = 2Q for 1 « w « L. This has to be compared with 
c, = 2QlnQ. 

8 Conclusion 

In this work we have developed in detail the SP formalism for coupled CSP. 
We find that the SP thresholds of spatially coupled random K-satisfiability 
and Q-coloring ensembles nicely saturate towards the SAT-UNSAT phase 
transition threshold of the individual ensemble. Moreover the SAT-UNSAT 
phase transition threshold of the coupled and individual ensembles are iden- 
tical as required by theorems [1] and [2l The saturation of the SP threshold is 
remarkably similar to the one of the Belief Propagation algorithmic threshold 
(towards the optimal one associated to the Maximum a Posteriori decoder) 
observed in coding theory. 

Let us point out a few issues that would deserve more investigations, and 
to which we hope to come back in the future. 

The large K and Q analysis has shown that when a is in a small inter- 
val where the zero-energy complexity is strictly positive, the warning and 
complexity densities form kink-like profiles. These are very similar to the 
kink-like magnetization and free energy densities found in the CW chain. A 
possible interpretation of the complexity density profiles is that the clusters 
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do not only have a "size" given by their internal entropy but also have a 
"shape" that could be taken into account by an extension of the entropic 
cavity method. The simplest system where this issue could be elucidated is 
the XOR-SAT system for which the clusters can be precisely defined [32]. 
We hope to come back to this question in the near future. 

As briefly discussed in the introduction, the entropic cavity method pre- 
dicts the existence of other thresholds, namely the dynamical and condensa- 
tion thresholds. We have checked that the condensation one is the same for a 
coupled and individual ensemble (for L — )■ -|-oo). This observation is consis- 
tent with the theorems of Appendix A. We also observe that the dynamical 
threshold of the coupled ensemble saturates towards the condensation one, 
for K and Q > 4. For K = "i the dynamical and condensation thresholds 
coincide already for the individual ensemble. We consistently observe that 
they remain unchanged by coupling. These observations deserve more in- 
vestigations, in particular the nature of the condensed phase, the freezing of 
variables, the behavior of correlation functions and the possible relevance of 
the shape of clusters. 

The present work could find applications in a new method for proving 
lower bounds on as (and possibly Oc). We hope that by choosing the right 
analyzable algorithms one may reach significant improvements of the best 
existing lower bounds. One requirement on the algorithms is that they should 
be able to propagate in the bulk the " seed " given by the reduced hardness of 
the coupled instances at their boundaries. We have seen that this is the case 
for simple peeling-type algorithms which are purely deterministic. Together 
with D. Achlioptas, we have observed that this is also the case for classic 
stochastic algorithms if they are augmented by a suitable scheduling |42j . 
Related ideas have been used recently within the context of a coupled CSP 
scheme for source coding [33] • A coupled low-density generator-matrix code 
is considered, and it is shown (numerically) that applying belief-propagation- 
guided-decimation with suitable scheduling, allows to approach the optimal 
rate-distortion curve of the individual code ensemble. 

A Free energy 

We sketch the proof of the finite temperature analogs of theorems [1] and El 
An important consequence is that the free energies of the coupled and indi- 
vidual ensembles have the same singularities in the (a,/3) plane (see flll4p ). 
This is consistent with the fact that the average ground state energies, and 
consequently the SAT-UNSAT thresholds, are the same. 

The Gibbs distribution (at "inverse temperature" /3) associated to the 
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coupled CSP Hamiltonian is 

fi^ix) = J-e-Z'^co.fe), z,,,, = e~^^-^^\ (107) 

■^cou ^ 

and the average free energy per node is 

/7v,L,«,(a,/3) = -^E[lnZeoJ. (108) 

The corresponding quantities 'H}?ou(x) are associated a chain to with periodic 
boundary conditions (see section [2]); these will be denoted by a superscript 
"per". Note that to get these quantities for the underlying system, one sets 
L = w = 1 in these definitions; the average free energy per node will be 
denoted by fN{a,f3). 

Remark about the entropy. The average entropy is defined as 

d 

SN,L,w{a, /3) = ■gjp^fN,L,w{(^, /3) = /3(E[('Hcou)] - fN,L,w{a, /?)) (109) 

where (— ) is the average with respect to (11071) . Theorems [3] and H] for the free 
energy have analogs for the average internal energy, and as a consequence 
also for the average entropy. Thus the entropy of coupled and individual 
ensembles have the same singularities in the (a, /3) plane. This is consistent 
with the observation that the condensation threshold at zero temperature is 
the same for both ensembles. 

Theorem 3 (Comparison of open and periodic chains). For general coupled 
CSP [N,K,a,w,L\ ensembles we have 

IW(«,/3)-/^lJa,/3)|<— . (110) 

Proof. We write (with the same notations than in the proof of theorem [T]) 

X x',x" 

and from fITB]) we deduce 

Applying — -^^log on each side of this inequality, we obtain the desired 
estimate. □ 
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Theorem 4 (Comparison of individual and periodic ensembles). For K-SAT 
and Q -coloring the limits limN-^+oo /^) ^'^^ liniTV-i^+oo fN^ct, P) exist, 

and are continuous in for all L. Moreover, 

1™ = lim fN{a,^). (113) 

Theorems [3] and m yield (recall limtherm = ^^^l^+oo^^^n^+oo) 

lim /Ar,L,»(a,/3) = lim f^^^a.P) = lim /Ar(a,/3). (114) 

therm therm ' ' iV— >+oo 

Proof. The proof of existence and continuity of limits for — )■ +oo {L 
fixed) is identical to [IB], so we do not repeat it here. The proof of the 
equality uses the same interpolating r-ensembles between the connected, ring 
and disconnected ensembles defined in subsection 15.21 The associated Gibbs 
measures, free energies and expectations will be denoted by scripts r, conn, 
ring and disc. 

By an argument similar to that of theorem [3] we have the analogs of 0791) , 

iDD, dHU), 

- limAr^_|_oo ^g^Econn [log ] = limAT^+oo /Ar(a,/3), for L fixed, 
■■3^Edisc[logZdisc] = /7v(a,/3) + 0(A^^^/2), uniformly in L, 
i^Ering[logZring] = ff;VJa,f3) + 0(iV-i/2), uniformly in L. 



Thus, it is sufficient to show that 

- -^Econn[l0g^conn] < - "^Ering [log ^ring] < " "^Ejisc [log ^disc] • (US) 

To prove these inequalities we will use the r-ensembles. It suffices to check 
the analogs of (1841) and (1^21) . namely for an intermediate graph G, 

- (E,[logZG. I G] - \ogZ^) < -(E,_i[logZG._, I G] - log%), (116) 

and then average over G. 

Consider the graph G and add a new constraint node n to it. The precise 
way in which n is connected to the variable nodes is deferred to a later stage 
of the argument. We have 

G x:n is UNSAT x:n is SAT 
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This is equivalent to 
Z, 



GUn 



^, l-(l-e-^) ^f^,G^^- (118) 

G zin is UNSAT 

Taking the log and expectation over n for a given G, we obtain 
-E[log^ I G\ = -E[log{l-(l-e-'^) Yl ^'^d^)} I (119) 

G x:n is UNSAT 

Note that the left hand side is identical to that of flll6p . To compute the 
expectation we expand — log(l — x) = ^i^i y , 

-E[log^|G]=$:^i^f^ 

G i>i 

xE[ Y /x,,c(^(^)).../i,,6(;^('))}|G]. 

xW,...,xW:n is UNSAT 

(120) 

The sum over "real replicas" x^^\ . . . is over assignments such that n is 
UNSAT for all / of them, so the expectation in fll20p equals 

-T Yl e-^^'>'=^^o^'''^^E[l{n\]mAT onallx^p\ h = l,...,l} \G]. 



(121) 

Up to this stage the arguments are completely general: they apply both 
to coloring and satisfiability. We specialize the rest of the proof to /^-SAT 
and leave coloring as an exercise. 

We first derive (11161) for the r-ensemble that interpolates between the 
connected and ring ensembles. This then implies the left inequality in (IllSp . 
Given G and given a term . . . , x*-'-* in (I12ip . let J-" be the set of variable 
nodes with frozen bits, i.e those variable nodes such that the bit takes the 
same value in all assignments x^^^ through x^^\ Below we will also need the 
sets J^z = ^ ^ Vz- When n is connected u.a.r. to the LN variable nodes we 
go from G to a, Gr graph and 



E, [l{n UNSAT on all x^p\ h = 1, . . . ,1} \ G] = ^ ( • 



(122) 



On other hand when n is first affected u.a.r. to a position z and then con- 

10 — 1- 

fc=0 



nected u.a.r. to the wN variables in UT}Vz+k mod l, we go from G to a Gr-i 
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graph and 

E,_i [l{n UNSAT on all x!-p\ h = 1, . . . J} \ G] 
-if ±(^TlT 

By convexity, the quantity in f ll22p is smaller than the one in (11231) . Using 
this fact together with (I119p . (11201) . (11211) . we obtain the final inequality 
(11161) . This implies the left inequality in (IllSp . 

The derivation of (I116p for the r-ensemble that interpolates between the 
ring and disconnected ensembles is similar. When n is first affected u.a.r. to 
a position z, and then connected u.a.r. to variable nodes in the set we 
go from G to a Gr-i graph. Thus, 

E^,_i [l{n UNSAT on all x!-p\ /i = 1, ...,/} | G] 




K 

(124) 



Finally we notice that by convexity, (I123p is smaller than (I124p , so that using 
again (I119p . (I120p and (I12ip we obtain the final inequality (I116p . This now 
implies the right inequality in (IllSp . 

□ 

We now turn to the the case of XORSAT which has to be treated some- 
what differently. All definitions of average ground state energies and free 
energies are the same as usual. 

Theorem 5 (Energy comparisons for XORSAT). For K -XORSAT with even 
K the limits lim^^+oQ In^l wi^^ f^) o,''T'd ^i^^N-^+oo fN^o:, (3) exist, are continu- 
ous in {a,P), and are equal. The same holds for the zero temperature quan- 
tities, i. e for limN^+oo e^N^L,wi<^) ^'^^ limTv^+oo eAr(a) . 

Proof. Existence and continuity of the limits for /T- XORSAT follows from 
sub-additivity which was already proven in in [17] for K eveij^. Here we 
concentrate on the equality of limits. The proof uses exactly the same inter- 
polations as in the proof of theorem |H 

*The argument in [TJ covers also A'-SAT for even K, but a small modification of it 
extends the proof to odd K; however for XORSAT with odd K it not clear how to extend 
the proof. Strangely enough, another case were such arguments break down is that of pure 
ferromagnetic diluted interactions. 
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First we prove the same relation as in (11151) for the case of XORSAT. For 
this we proceed exactly as in equs. flll6p - fll21l) and reduce the problem to 
estimating the expectation 

E[l{n UNSAT on all x^"), p = 1, ...,/} | G] , (125) 

according to the various ways of connecting the new constraint node n. It will 
be useful to represent the indicator function in an algebraic wajl^. Suppose 
that n connects to variable nodes ni, ■ ■ ■ , Uk, then 

/ K 

l{n UNSAT on all x^p\ p = 1, . . . , /} = JJ - (l - 6„ JJ(-l)^"'^) 

p=l v=l 
0<r<l {pi,...,pr}C{l,...,l}v=l^ ^ 

When we take the expectation over bn ~ Bernoulli (1/2) only the terms with 
r even remain, 

^ r even ^ C 1 

E n{(-i)*"---(-i)*'|- (127) 

0<r<l {pi,...,pr}C{l,...,l}v=l^ ^ 

Now, it remains to compute the rest of the expectation on possible ways of 
connecting n. We define "local overlap parameters" 

g(P....,.) = i_^(_l).<-\..(_l)e>. (128) 
1=1 

Let us first consider the interpolation between the ring and fully connected 
ensembles. To go from G to Gr we connect n u.a.r. among all LN variable 
nodes v = {i, z). Thus ( I125p becomes 

L 

r even f -, 2 \ K 

^ E E li E Q'r ■ (129) 

0<r-<i{pi,..,p,}c{l,...,0 ^ z=-^+l ^ 

On the other hand to go from G to Gr-i, we first affect n to a position z u.a.r. 
and connect its K edges to variable nodes v = [i, z + k) with k G {0, . . .w — 1} 
u.a.r.. This time (I125p becomes 

^ r even ^ 2 f w—1 K 

fE E i E {^EeriJ . (130) 

0<r<i{pi,...,p4c{l,...,0 z=~^+l^ k=0 ^ 

^The method used here can be used also for satisfiabihty and coloring and although it is 
somewhat longer, it may be useful when it is not obvious how to define "frozen variables" . 
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Convexity of the function for even K implies that fll29p < fll30p . which 
then imphes the left inequality in f lllSp . Unfortunately at this point the 
argument breaks down for odd K because we do not control the sign of the 
overlap parameters. 

Consider now the interpolation between the ring and disconnected en- 
sembles. When the extra node is first affected to z u.a.r. and its K edges 
connected u.a.r. to the variable nodes at the same position, we obtain for 
([125]) 

L 

-. r even 2 f 

0<r<l {pi,...,Pr}C{l,...,l} 2=-|+l 

Again convexity of for K even implies that (I130p < (ll3ip . which then 
implies the right inequality in (IllSp . 

We have proven (IllSp for any finite /3, and since L and are finite, there 
is no difficulty in taking the /3 — )■ +00 limit. This yields the zero temperature 
version of this inequality, namely (182|) applied to XORSAT. 

Finally with the help of (182|) and (IllSp we conclude (proceeding as in 
the previous proofs) that the average ground state and free energies of the 
individual and periodic ensemble are equal in the limit A^ +00, with L 
and w fixed. 

□ 

B Review of the cavity method and survey 
propagation equations 

The main assumptions of the cavity method draw on the concept of pure 
(or extremal or ergodic) state. While this concept can be given a rigorous 
meaning for "simple" models [28], [29], it still forms a heuristic framework 
in the context of disordered spin systems. We refer the interested reader to 
[TT] . [33], [3S], [Snj, [SZ] for more information and various approaches. 

Infinite volume Gibbs measures form a convex set whose extremal points 
play a special role and are called pure states. A crucial property of a pure 
state is that the correlations decay (usually exponentially fast) with the graph 
distance. This is not true for non-trivial convex superpositions of pure states. 
For "simple" Ising-type models the number of pure states is "small" and 
they are related by a broken symmetry. Disordered spin systems can have an 
exponential (in system size) number of pure states and the broken symmetry, 
if only there exist one, is hard to identif}0. The growth rate of the number 

^''Within the replica formalism it is a formal symmetry between " a number" of copies 
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of pure states, is called the complexity. This is a notion analogous to the 
Boltzmann entropy, but at the level of pure states, instead of microscopic 
configurations, for which one develops a new " level" of statistical mechanics. 

We assume that this picture can be taken over to CSP and even coupled- 
CSP. Let p index the set of pure states (we called them pure Bethe states in 
section [2]). The special feature about systems on random graphs is that they 
are locally tree-like with high probability. Thus, since for each pure state p 
the correlations decay sufficiently fast, the marginals of the pure state p can 
be computed from the sum-product (or Belief Propagation) equations 



xa(cz)\iu jved{cz)\iu 

'^tlM= n ^tl^ui^^u)■ (133) 

bv(^d{iu)\cz 

In 01321) . fll33p = means that the right hand side has to be divided by a 
normalization factor to get a true marginal on the left. The free energy of 
the pure state p is given by the Bethe expression, 

+E'4e n 

iu ^ Xiu cz£d{iu) 

- E ^4T.^lu-.cz{^^u)oi:i,M]- (134) 

{cz,iu)£E ^ Xiu ^ 

To investigate the zero temperature limit (3 — )■ +00 we set 



(135) 

When /3 — )■ 00, the sum-product equations (11321) and (I133p reduce to the 
min-sum equations 

Eiu^cz(^Xiuj miU'^l, ^ ^ Ei)y^i^{Xiu) Ciu^cz\ 

bv(id{iu)\cz 

\{Ebv^iu}hv&d{iu)\cz~\^ (136) 



of the system. 
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Ecz^iu{xiu) = mm {(1 - ijJczixaicz))) + ^ Ej^^^zixj)} - Ccz^iu 

jvGd{cz)\iu 

= Gcz~^iu\_{Ejy^(.2:}jv€d{cz)\iii\- (-'-37) 



Here, Ciu^cz and Ccz^iu are normalization constants fixed so that 
min^,„ Eiu^cz{xiu) = min^,^ Ecz^iu{xiu) = 0. The Bethe formula for the free 
energy of a pure state reduces to an expression for its ground-state energy 

hm = S[{El:lM, Eit^u{■)}l (138) 

/3— s>H-oo 

where the functional S is given by 
S[{E 

iu^czi Ecz^iu 

}] = y min{(l - ipczixo(_cz) ))+ >. ^» 

cz iu(^d{cz) 

+ ^min{ ^ Ecz^iu{xiu)} - ^ min{E 

lu czGaiu {cz,iu) 

cz iu 

^ ^ ^cz,iu [{-^m— >-C2! -^cz— • (139) 
(cz,m} 

We assume that the heuristic low temperature picture carries over to the zero 
temperature case. In this context pure states become clusters (in Hamming 
space) of minimizers of the Hamiltonian. Each cluster is characterized by 
a set of messages {E^j^_^^^{.) , E^J^^^^{.)} . At zero temperature, two minimiz- 
ers belonging to the same cluster can be connected by successive flips with 
infinitesimal energy cost, while for two minimizers belonging to different clus- 
ters this is not possible. 

Now we wish to compute the complexity ([Hj) which counts the number 
of clusters. For this we introduce a generating function 

S(y) = e-J'^[^-^--"(-)'^-— (-^^l . (140) 
p 

When y — t- +oo the sum is dominated by solutions of the min-sum equations 
with minimal Bethe energy. This object can be viewed as a partition function 
for the effective Hamiltonian (11391) at inverse "temperature" y (the so-called 
Parisi parameter). Now, if we take a in the SAT phase the minimum Bethe 
energy vanishes and the complexity fll4p is given by 

T.L,M= lim lim ^lnS(7/). (141) 
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A negative complexity signals that there are no zero energy states and that 
the system is in an UNSAT phase. When this happens one has to generalize 
these formulas to allow for an energy dependent complexity (obtained by the 
Legendre transform of lnS(?/)) but this aspect will not concern us here. For 
CSP's it can be shown that the min-sum messages take discrete values in a 
finite alphabet. Therefore we have 

xl[L-y'- n ^E,,^^u = gcz^^u)\. (142) 

cz ^ iu€d{cz) ^ 

The arguments of the functionals i^ial—], ^cz[— ], ^m,cz[— ] and Qiu^cz[—], 
Qcz^iu[-\ are the messages {Eiu^czi,-), Ecz^iu{.)}] they are not exphcitly writ- 
ten to ease the notation. It can easily be seen that this is the partition func- 
tion of a new graphical model which is still sparse. Edges ((c, 2;), {i,u)) now 
correspond to degree two "constraint" nodes, and nodes (c, z) and {i, u) now 
correspond to "variable" nodes. Therefore f ll4ip can be computed from the 
Bethe approximation for this new model. The underlying assumption here 
is that the new effective model has a unique "pure state" with fast decaying 
correlations. This is called the level- 1 cavity method. If this assumption 
breaks down, one should repeat the whole scheme, obtaining a level-2 cavity 
method (and so on). At level-1, the Bethe approximation can be expressed in 
terms of new beliefs - called surveys - Qiu^cz{,Eiu^cz{,-)) and Qcz^iu{Ecz^iu{.)) 
that count the fraction of clusters p for which eI^I^^^{.) = Eiu^czi-) and 
eI^\-^{.) = Ecz-^iu{-)- Note that these are the messages on the induced 
graph obtained by eliminating the degree two constraint nodes of the new 
model. We have 

lnH(y)=5^1n| Yl n ^»--| 



iu I R \ ^a, ^ cz&diu 



CZ-^IU 



5^1n| Y e-y'--Q,u^,-Mcz-.iX (143) 
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The messages satisfy the survey propagation equations 

X Y\. Qbv^iu{Ebv-yiu), (144) 
bv(^d{iu)\cz 

Qcz-^iuiEcz^iu) = ^ ] 1 (-E'cz^m = Gcz^iu)(i 3/ cz-mi 

{-E'ju^cz}jt,ge(cz) 

X n Qjv^cz{Ejv^cz)-, (145) 
jj;€0(c2)\jji 

where again = means that the right hand side has to be normahzed. 

In the SAT phase one takes y — )■ +00 in order to compute the complexity. 
This has the effect of reducing the sums in fll44p . fll45p and fll43p . to surveys 
such that Ciu^Qz ~ C^z^iu ~ and S^z — £iu ~ ^iu,cz ~ 0. 
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